THE BOUNDED SPHERICAL FUNCTIONS FOR THE FREE 
TWO-STEP NILPOTENT LIE GROUP 



VERONIQUE FISCHER 



Abstract. In this paper, we give the expressions for the bounded spherical 
functions, or equivalently the spherical functions of positive type, for the free 
two-step nilpotent Lie groups endowed with the actions of orthogonal groups 
or their special subgroups. Next we deduce some results about the (Kohn) 
sub-Laplacian, and we compute the radial Plancherel measure. 



1. Introduction 

A (connected, simply connected) nilpotent Lie group which forms with a compact 
Lie group a Gelfand pair, is at most of step two, and the bounded spherical functions 
are the spherical functions of positive type [T] . The cases of the Heisenberg group 
with some subgroups of the unitary matrix group are well known [2J, and the 
bounded spherical functions are then explicit. In this paper, we are interested 
by the Gelfand pair formed by the free two-step nilpotent Lie groups with the 
actions of orthogonal groups. The expressions of some of the corresponding bounded 
spherical functions were given in [ITJ Section 6] with a sketched proof. Here we 
give the expression of all such functions with a complete proof, and we obtain the 
corresponding eigenvalues for the sub-Laplacian and the radial Plancherel measure. 

This paper is organized as follows. After recalling some definitions and setting 
some notations, we give in the second section the statement of the main results: 
the expressions of the bounded spherical functions on the free two-step nilpotent 
Lie groups. In the third section, we recall a few facts about spherical functions and 
representations, which allow to construct our bounded spherical functions in the 
following section. We also give an equivalent method of construction from which 
we obtain some properties of the sub-Laplacian and the radial Plancherel formula 
in the fifth section. 

We shall omit some computations and the proof for the case of the special or- 
thogonal group. We refer the interested reader to the French thesis of the author 
0. 



2. The Free Two-step Nilpotent Lie Groups 

Here we give definitions and notations for the free two-step nilpotent Lie groups 
and algebras; we also present the action of orthogonal groups. 
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First Definition. Let Afp be the (unique up to isomorphism) free two-step nilpotent 
Lie algebra with p generators. The definition using the universal property of the 
free nilpotent Lie algebra can be found in Chapter V §4]. Roughly speaking, 
Af p is a (nilpotent) Lie algebra with p generators X±, . . . , X p , such that the vectors 
X±, . . . , X p and Xi j = [Xj, Xj], i < j form a basis; we call this basis the canonical 
basis of Afp, 

We denote by V and Z, the vector spaces generated by the families of vectors 
Xi, . . . , X p and X^j := [Xi, Xj], 1 < i < j < p respectively; these families become 
the canonical base of V and Z. Thus Af p = V © Z, and Z is the center of Af p . With 
the canonical basis, the vector space Z can be identified with the vector space of 
antisymmetric p x p-matrices A p . Let z = dim Z = p(p — l)/2. 
The connected simply connected nilpotent Lie group which corresponds to Af p is 
called the free two-step nilpotent Lie group and is denoted N p . We denote by 
exp : Af p — >■ N p the exponential map. 

In the following, we use the notations X + A E Af, exp(A + A) £ N when X G 
V,Ae Z. We write p = 2p' or 2p' + 1. 

A Realization of Af p . We now present here a realization of Af p , which will be helpful 
to define more naturally the action of the orthogonal group and representations of 
N p . 

Let (V, <, >) be an Euclidean space with dimension p. Let 0(V) be the group of 
orthogonal transformations of V, and SO(V) its special subgroup. Their common 
Lie algebra denoted by Z, is identified with the vector space of antisymmetric 
transformations of V. Let Af = VffiZ be the exterior direct sum of the vector 
spaces V and Z. 

Let [, ] : V x V —> Z be the bilinear application given by : 

[X,Y].(V) =< X,V > Y- < Y,V > X X,Y,V eV . 
We also denote by [, ] the bilinear application extended to Af x Af — > Af by: 

[•! ]My.Z = [-, -}zxAf = . 

This application is a Lie bracket. It endows Af with the structure of a two-step 
nilpotent Lie algebra. 

As the elements [X, Y), X,Y € V generate the vector space Z, we also define a 
scalar product < , > on Z by: 

< [X,Y], [X',Y'\ > = < X,X' ><Y,Y' > — < X,Y' >< X',Y >, , 

where X,Y,X',Y' e V. 

It is easy to see as a realization of Af p when an orthonormal basis X\ , . . . , X p of 
(V, <, >) is fixed. 

We remark that < [X,Y], [X',Y'] >=< [X,Y]X',Y' >, and so we have for an 
antisymmetric transformation A 6 Z, and for X, Y 6 V: 

(1) < A, [X, Y]> = < A.X, Y > . 

This equality can also be proved directly using the canonical basis of Af p . 
Actions of Orthogonal Groups. We denote by 0(V) the group of orthogonal linear 
maps of (V, <, >), and by O p the group of orthogonal p x p-matrices. 
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On Af p and N p . The group 0(V) acts on the one hand by automorphisms on V, 
on the other hand by the adjoint representation Adz on Z. We obtain an action 
of OiV) on Af = V ® Z. Let us prove that this action respects the Lie bracket of 
Af. It suffices to show for X, Y, V G V and k € O(V) : 

[k.X,k.Y](V) = < k.X,V > k.Y- < k.Y,V > k.X 
= k. (< X* k.V > Y- < Y* k.V > X) 
= fc.[X,Y](fc _1 .V) = Ad z k.[X,Y] . 

We then obtain that the group 0(V) (and also its special subgroup SO(V)) acts by 

automorphism on the Lie algebra Af, and finally on the Lie group N. 

Suppose an orthonormal basis Xi,...,X p of (V, <, >) is fixed; then the vectors 

Xij :— [Xi,Xj],l < i < j < p, form an orthonormal basis of Z and we can 

identify: 

• the vector space Z with A p , 

• the group 0(V) with O p , 

• the adjoint representation Adz with the conjugate action of O p on A p : 
k.A = kAk^ 1 , where k G O p , A G A p . 

Thus the group O p ~ 0(V) acts on V ~ l p and Z ^ A p , and consequently on 
Af p . Those actions can be directly defined; and the equality [kX,k.Y] = k.[X,Y], 
k £ O p , X, Y £ V, can then be computed. 

On J^p. Now we describe the orbits of the conjugate actions of O p and SO p on A p . 
An arbitrary antisymmetric matrix A 6 _4 p is O p -conjugated to an antisymmetric 
matrix D 2 (A) where A = (Ai, . . . , \ p >) G M. p ' and: 



(2) A.(A) 







X p 'J 



(0) 



where J 



((0) means that a zero appears only in the case p = 2p' + 1.) Furthermore, we can 
assume that A is in C, where we denote by C the set of A = (Ai, . . . , X p i) G W , 
such that Ai > . . . > A P ' > 0. 

An arbitrary antisymmetric matrix A G A p is S'Op-conjugated to 

£>!(A):=D 2 (Ai,...,V-i.eV) , 
where e = ±1 and A = (Ai, . . . , X p >) € C. 



3. Notations and Main Results 

We give here the notations for special functions that will be used to present the 
main results of this paper. First, we recall the definitions of the Bcsscl and Laguerre 
functions, and we set some notations for parameters. We give then the expression 
of the bounded spherical functions. 

Notations for special functions. We will use the following well known functions : 

• the Gamma function T, 

• the Laguerre polynomial of type a and degree n: L" [TH §5.1], 

• the Bessel function of type a: J a [12, §1.71], H ch. II, 1.1]. 



4 



Let us now define the normalized Laguerre function £ n , Q — £„, Q /C™ +Ct where: 
£ nia (x):=L%(x)e-% and C% +a = , 

and the reduced Bessel function by: 

J a (z) :=r(a + l){z/2y a J a (z) . 

Let n = 1,2,..., and let dk denote the Haar probability measure of the compact 
group K = O n or SO n . If <,> denotes the Euclidean scalar product, and |.| the 
Euclidean norm of R n , we recall for any fixed xo € K n such that |xo| = 1 [H ch.II, 
1.1]: 

(3) Js=i(\x\) = I e i<kx ' Xo> dk . 



Jk 

Parameters. To each A G C, we associate: po the number of Ai 0, p\ the number 
of distinct \ ^ 0, and fix, ... , fi Pl such that: 

(4) {fn > fi 2 > • ■ ■ > M P1 > 0} = {Ai > A 2 > . . . > A po > 0} . 
We denote by mj the number of Aj such that A, = fij, and we put: 

(5) mo := m' Q := and for j = 1, . . .px rrij :— mi + . . . + mj 

For j — 1, . . .p\, let pr^ be the orthogonal projection of V onto the space generated 
by the vectors Xzi-i, X 2 i, for i = m'j_ x + 1, . . . , m'j. 

Let M. be the set of (r, A) where Ae£, and r > 0, such that r = if 2po = 
Expression of the bounded spherical functions. The bounded spherical functions of 
(N p ,K) for K = O p or SO p , are parameterized by 

• (r, A) € A4 (with the previous notations po, pi, Hi, Wj associated to A), 

• I e W 1 if A ^ 0, otherwise 0, 

• e = ±1 if if = SO p , otherwise 0. 

Let (r, A), I and e be such parameters. If A ^ 0, we define the function r > A ' ; > e by: 

/r,A,i,6A \ _ / c,r,A,!,e 



(7) e^(exp(X + A)) - e i<rX Pl X> ei <D5(A),A> n£ ( ^| pr . W | 2 ) . 

i=i 2 J 

If A = 0, wc define the function <fi r, ° by: 

(8) <f'°(n) = Js^(r\X\) , n = exp(X + A) e N p . 

In Section [SJ we shall prove the following result in the case K — O p (the case 
K = SOp is similar and can be found in [S]): 

Theorem 3.1. The bounded spherical functions of (N p , K), for K = SO p or K = 
O p , are the functions (ff- A ^^ given by $6$ and f3J), where (r,A) € Ai and I G N Pl if 
A ^ 0, and e = ±1 j/if = SO p , e = if K = O p . 

In Section [6l we shall also express r '° and r ' A,/ in terms of representations of 
N p and obtain their eigenvalues for the sub-Laplacian and the radial Plancherel 
measure. 



(6) 4> r ' A ' l ' e (n) = / O r ' A < l ' e (k.n)dk, neNp 

where O r < A > l > e is given by: 
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4. Spherical Function and Representation 



In this section, we recall some of the properties of spherical functions, Gelfand 
pairs and representations, which will be used in the proof of Theorem 13. II 

In this article, we use the following conventions. 
The semi-direct product K x N of two groups K and N such that K acts on TV by 
automorphism, is defined by the law: 

(fcx,ni), (&2,7i2) S K K N, (fci, rii).(fc 2 , n 2 ) = (fcifc 2 , ni fci.n 2 ) . 

All the groups are supposed locally compact, second countable and separable, and 
their continuous unitary representations on separable Hilbert spaces. 
For such a group G, we denote by G the quotient set of the irreducible represen- 
tations by the equivalence relation ~. We often identify a representation with its 
equivalence class. 

Definitions and properties. Let if be a compact group, which acts continuously on 
a group N . Let dn be a Haar measure on N, and dk the normalized Haar measure 
on K. We assume that dndk is a Haar measure on the group G — K x N , and that 
this group is unimodular. 

Let (N) be the set of continuous compactly supported if-invariant functions on 



A Jf-invariant function <f> on N is spherical on N if for all f,g € (N) we have: 



If is a spherical function on N for K, the function $ on G = K x N given by 
n) = 4>(n) is also called a spherical function on G for K. 

Remark 4.1. Suppose K and N are Lie groups and G = KG° , where G° is the 
connected component of the neutral element. Then the spherical functions $ on 
G are analytic and they are the common eigenfunction of (G-)left and K -invariant 
differential operators on G, such that $(0) = 1. Equivalently, the spherical function 
(j) on N are analytic and they are the common eigenfunction of (N-)left and K- 
invariant differential operators on N, such that 0(0) = 1 [6, ch.X]. 

As examples of spherical functions, we shall provide their expressions on Heisen- 
berg groups. These will be used during the proof of Theorem 13. II 
If C^(N) is a commutative algebra for the convolution product, then (N,K) is 
called a Gelfand pair. 

We recall the link between bounded spherical functions and representations, which 
we will use to construct our bounded spherical functions: 

Theorem 4.2. Let (N,K) be a Gelfand pair. 

a) [6l ch.X], [31 ch.IV,I] The vector space of K -invariant vectors of an irre- 
ducible representation on G = K x N is of dimension at most one. 

The spherical functions of positive type (on G) are the positive definite 
functions $ (on G) which are associated to an irreducible representation 
with at least one non zero K -invariant vector. 

For the representation associated to a positive definite function, the vec- 
tor space of K -invariant vectors is 

b) [U Corollary 8.4] If N is a nilpotent Lie group, then the bounded spherical 
functions are the spherical functions of positive type. 



N. 




6 



It is known p] Theorem 5.12] that (N p ,SO p ) and consequently (N p ,O p ) are 
Gelfand pairs. Thus to obtain the bounded spherical functions of (N p ,O p ), we 
need to describe classes of representations of G :— O p tx N p . In this section, we 
shall compute those of N p by the orbit method (see [lO] or [8]). We compute G 
using Mackey's Theorem [9l ch.III B Theorem 2], provided that we describe N p /G. 
To describe N p , the classes of representations of N, we shall use the orbit method 
(see [TO] or For a connected simply connected nilpotent Lie group N, we will 
denote by Tf the classes of representation of N associated to / G Af* . First, we set 
the following conventions for elements of Af* . 

Conventions regarding elements of Af* . In this section and in the rest of this paper, 
we write N = N p , its Lie algebra Af p = Af and the dual AT* — Af* . We denote by 
V* and Z* the dual spaces of V and Z respectively, and by XJ, . . . ,X* the dual 
basis of X\, . . . , X p . 

Let A* E Z* be identified with an antisymmetric transformation (by the scalar 
product on Z). We associate to it the bilinear antisymmetric form u>a* on V, given 
by: lo a , (X, Y) =< A*X, Y >, X, Y G V. The radical of u A , coincides with ker ,4*; 
and its orthogonal complement in (V, <, >) is 3^4*, the range of A*. So on "3A*, 
u>a* induces a simplectic form oja*,t and the dimension of QA* is even and will be 
denoted by 2pQ. 

Suppose we have fixed E\, a maximal totally isotropic space for 0JA*,r- Then E 2 — 
A* E\ is the orthogonal complement of E\ in (Sj4*,<,>) and a maximal totally 
isotropic space for uiA*,r- The dimension of E\ and E2 is pq. We denote by qo : 
V — > ker A* , qi : V — > E\ and q 2 : V — > E 2 the orthogonal projections. 
Description of N p . Now we describe N p , the classes of representations of N (we 
will only need some of these classes). 

We need to describe first the representatives of Af* /N. The co-adjoint representa- 
tion is given for n = exp(X + A) E N by: 

X*eV*,A*eZ* Coad.n(X* + A*) = X* + A* - A*.X . 

We can thus choose the privileged representative X* + A* (X* E V and A* E Z), 
of each orbit Af* /N, such that X* E ker A*. Let / = X* + A* have this form. We 
define the bilinear antisymmetric form on Af associated to /: 

VV,V'EAf : B f (V,V) = f([V,V']) ■ 

Because of (P), we have: 

B f (X + A,X' + A') = f([X,X'}) =<A*,[X,X']>= w A »{X,X') . 

Some easy computations show that a polarization Cf at / and an associated rep- 
resentation (Hx*,a* j Ux*,A*) are given by: 

• if A* = 0, then Bf = 0, £/ = Af, and Ux*,A* is the one dimensional 
representation given by: exp(X + A) n- exp(i < X*,X >). 

• if A* (with the previous conventions about A* E Z*\ we assume 
that we have chosen a maximal totally isotropic space E\ for u>A'.n and 
so Cf := E2 ® ker^l* © Z is a polarization at /. Another choice for E\ 
gives another polarization at /, but does not change the class of Ux-.A'- 
We compute Hx*,a* = L 2 (Ei), and for F E Hx*,A*, n = exp(X + A), 
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X' G Si: 

C/x.,^(n).F(X') = exp^< A* ,^[ qi (X + 2X'),q 2 (X)} + A > 
(9) e J<x *< x> F( gi (X)+X') . 



) 



Kirillov's Theorem gives: 



Proposition 4.3. For A* <s Z* , and X* e ker A* cV, we ftcwe: 



U x »,a» € Tx»+A* 



Furthermore, when A* and X* ranges over Z* and ker A* respectively, Ux*,a* 



where (X*) is the orthogonal space of X* in (V, <, >), and (A*) is the orthogonal 
space of A* in (Z,<,>). 

Remark 4.5. The restriction ofUx*,A* on Z is given by: 



Consequences of Kirillov's Theorem. Here we give simple consequences of Kirillov's 
Theorem, which will permit us to describe N p /G (where G = K k N p ). 
In this paragraph, N is a connected simply connected nilpotent Lie group, and G a 
group which acts continuously by automorphisms on N . We denote by Af the Lie 
algebra of N, and by TV* the dual of Af. Then G acts on N: 



For g £ G, we compute: g.Tf = T g j. We deduce: 

Corollary 4.6. The Kirillov map induces a one-to-one map from (Af*/N)/G onto 
N/G, which maps the G -orbit of f G Af* to the G -orbit ofTf. 

Under the previous hypothesis, for p e N, we denote its G-stability group by: 

G p = {g e G; g.p = p} . 
By Kirillov's orbit method, it is easy to see that: 

Proposition 4.7. Let N be a connected simply connected nilpotent Lie group, and 
K a group which acts continuously by automorphisms on N . Let G = K K N . 
We have (Af*/N)/G - Af*/G. 

Furthermore, let p € N be fixed. We may assume that p = Tf,f € Af* . Then 
the G-stability group G p is K p K N, where K p is the K-stability group of p, or 
equivalently of the N -orbit N.f of f : 



ranges over a set of representatives of each class of Nj 
Remark 4.4. The Lie algebra ofkevUx*,A* is: 




exp A h> exp(i < A*, A >) 



geG,peN g.p:=n^p(g 1 .n), 
and by automorphisms on the vector space Af* : 

geG,feAf* g.f := fig' 1 .n) . 



K p := {keK: k.p = p} = {k e K C G : k.f e N.f} 
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Bounded Spherical Function on the Heisenberg Group. Here, as example of spher- 
ical functions, we provide the expressions of the bounded spherical functions on 
Heisenberg groups for some compact groups. This will be used during the proof of 
Theorem 13.11 

We use the following law of the Heisenberg group H Po : 

Vft = {z u . . . ,z po ,t) , ti = (zj, . . . , z' po ,t') £ HP = C P0 x R 

1 Po 

h.ti = (zi+z[,..., z P0 + z' po , t + t' + - ^2 SziZi) ■ 

The unitary po x p matrix group U Po acts by automorphisms on W°. Let us 
describe some subgroups of U Po . Let pa,pi £ N, and m = (mi,...,m Pl ) £ N Pl 
be fixed such that Yl^Li m j = Po- We define m'^ for j = l,...pi by (O. Let 
K(m;pi;po) be the subgroup of U Po given by: 

(10) K^m-p^po) = U mi x ... x U mpi . 

The expressions of spherical functions of (H Po , K(m;pi;po)) can be found in the 
same way as in the case m = (po),Pi = 1 i-e. K = U po [U ch.V,II.6] using 
Remark |4. 11 here, we admit [5]: 

Proposition 4.8. (H Po , K(m;pi;po)) is a G elf and pair. 
Its bounded spherical functions on W° are: 

(1) u) = u\ t i with AeR* andl = (h,.. . ,l Pl ) £ W 1 : 

uj( Zl) ...,z P0 ,t) = e lXt .n£, jimj -i(^ XJ l^l 2 ) > 

(2) w = w,, with p = (pi, . . . , p pi ) and pi > : 

cj(z,t) = nj^im I J2 SP) ■ 

V m' J _ 1 <i<m' J 

During the proof of Theorem 13.11 we will use the following notations. To a 
spherical function ui for the Gelfand pair (H Po , K(m;po;pi)), we associate the cor- 
responding spherical function SI" on Hheis — K(m\po\pi) xH Po , and the irreducible 
representation (% W ,II W ) on H} le i S associated with 11". We compute easily: 

(11) iiw = u) X ,i n w (0,t) = exp(iAt) 

(12) Huj = uj^ n w (o,*) = 1 . 

5. Expression of the Bounded Spherical Functions 

This section is devoted to the proof of Theorem 13.11 for K = O p . Let G be the 
group K k N, where N = N p and K = O p . We fix the Haar measure dkdn on G. 
Overview of the proof. For p £ N, we denote by: 

• G p the G-stability group of p, 

• G p the set of v £ G p such that v\n is a multiple of p, 

• G p the set of v £ G p such that the dimension of the space of ifp-mvariant 
vectors is one. 
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By Mackey's Theorem [H ch.III B Theorem 2], when p and v range over a repre- 
sentative of each class of N and G p respectively, the representation induced by v 
on G gives a representative of each class of G. 

Because of the subgroup and intertwining number Theorems [9j ch.II A, Theorem 
1 and Lemma 5 respectively] , we easily get for v € G p : 

v G G p v\k p contains exactly one times 1k p 

(13) the space of K -invariant vectors of Ind G ^isaline. 

The proof of Theorem 13.11 is based on the two theorems and proposition which 
follow. We will explain after their statements how we deduce from them the ex- 
pression of all bounded spherical functions. 

First, we express the bounded spherical functions in terms of representations p G 
N/G and v G G p : 

Theorem 5.1. Let p G N and (W ' ,u) G G p . Then because of JTty . Indg p v G G 
has also a (non-zero) K -invariant line and the associated bounded spherical function 
is the function <jf given by : 

(14) <f>"(n) = J (v(I,k.n).u u , u v ) n „dk, neN , 
where u v is any unit K p -invariant vector. 

Furthermore, we obtain all the bounded spherical functions as <p u when p and v 
range over a set of representatives of N/G, and G p respectively. 

Next, to obtain all representations p G N/G, we describe Af*/G (see Corol- 
lary 

Proposition 5.2. Let 0(r,A) = G.(rX* + D 2 (A)) C N* . 
Then the mapping 

M -> Af*/G 
(r,A) ^ 0(r,A) 

is a bijection. 

Now, we describe G p , where p is a representation associated (by Kirillov) to a 
linear form on Af, which is a privileged representative of a G-co-adjoint orbit (just 
given in Proposition I5.2[) : 

Theorem 5.3. Let p G T f where f = rX* + D 2 (A) and (r, A) G M. 

a) //A = 0, G p — {v r '°}. The spherical function (jf which is associated 
(by (T^) tov = v r '° is (/) r ' a (given by 

b) If A ^ 0, G p G {v r ' A ' 1 , I G N Pl }. Each representation v = v r ^ 1 G G p has 
a K p-invariant line, and the spherical function <jf associated (by \1J$ ) is 
cf) r ' A ' 1 (given by 

The representations v r '° and i/ r ' A > 1 will be described during the proof (see p7|) 
and p8)» . 

For the moment, we will admit these two theorems and the proposition, and keep 
their notations. From Corollary 14.61 and Proposition ^. 2[ we deduce that: 

N/G = {T rX;+D2(A) , (r, A) G M} . 
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Under Theorems 15.11 and 15. 3[ the spherical bounded functions are the functions 
r '°, when r € R+, and cjf-^ 1 when (r, A) eMandie W 1 . 

If we prove Theorems l5.ll and l5.31 and Proposition l5.21 Theorem l3.1l will follow. The 
rest of this section will be devoted to this. We start with the proofs of Theorem l5.ll 
and Proposition 15.21 Then for a representation p € 7 1 r x*+D 2 (A)i we describe its 
G-stability group and the quotient group N = N/ ker p. We finish with the proof 
of Theorem 15.31 

Set G p . The aim of this paragraph is to prove Theorem 15.11 

Let p G N be fixed. Under Proposition 14.71 the G-stability group of p is G p — 
K p k N, where K p is the if -stability group (which is a compact subgroup of K). 
We fix the normalized Haar measure dk p on K pi and the Haar measure dk p dn on 
Gp. 

We fix (W,v) € G p and a unit if p -invariant vector u = u v e 'H 1 '. We denote by 
("H n ,II) the induced representation Indg v of v: 

Vg,g'eG feU u : = f(g'g) , 

and by / the function on G given by : 

f{k,n) = v(I,n).u, (k,n) £ G ; 

the vector / G "H n is if-invariant and of norm one. We can then associate to IT 
and / the bounded spherical function <f) v : 

r(g) - <n( 3 )./,/) wn , geG . 

We can easily obtain for g = (k, n),g' — (k' , n') G G: 

n(g).f(g') = v{I,n')u{I,k'.n).u , 
{Tl(g).f(g')J(g')) Hl , = (p(I,k' .n).u , u) n „ . 

We thus obtain the formula (fT4|) . 

We can now complete our proof of Theorem 15.11 Under Mackey's Theorem and 
property (|13p . when p and v range over a set of representatives of N/G and G p 
respectively, we get all the irreducible representations IT = Ind^ v having a K- 
invariant line. Under Theorem 14.21 the positive definite functions (jf associated to 
IT give all the bounded spherical functions. 

Description of J\f* /G. Here we prove Proposition 15.21 We easily compute for g — 
(k, n)€G with n = exp(X + A) e N and X* £ V*, A* e Z*: 

(15) Coad.g(X* + A*) = k.X* + k.A* - (k.A*).X . 

Let O E Af*/G be a fixed orbit. We associate to it A e C such that all the 
antisymmetric matrices AJ, where / = Xj + Aj g O, are if-conjugate, and K- 
conjugated to Z?2(A). 

Let / = Xj + A*f € O be fixed. We make the following choices: 

(1) let fc € K be such that k .A*j: = D 2 (A); 

(2) let X € V be such that {k .A*j).X e V* is the orthogonal projection Xq 
of k .Xj e V* on the kernel kerfc -^/ = kerD 2 (A); in particular, Xq = 
HQD 2 (A) = V; 

(3) let k' E K be such that k' .X e 3?L» 2 (A) for all X E 9?D 2 (A) and k' X* = 
rX* r € M+. 
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We get (k' k ,expX ).f = rX; + D 2 (A). 

We remark that QD 2 (A) = V is equivalent to p = 2po and Xi 7^ 0, i = 1, . . . ,p'. 
Proposition 15 . 21 is thus proved. 

Stability Group K p . The aim of this paragraph is to describe the stability group 

K p Of p G T rX * + D 2 (A)- 

Before this, let us recall that the orthogonal 2n x 2n matrices which commutes with 
^2(1, . . . , 1) (see <J2J) for this notation) have determinant one and form the group 

(n) 

Sp n n O n . This group is isomorphic to U n ; the isomorphism is denoted ip\ , and 
satisfies: 

Vk,X : ^\k.X) = ^\k)4^{X) , 
where ipc is the complexification: 

ip( n '(xi,yi; . . . ;x n , y n ) = (xi + iyi,...,x n + iy n ) . 
Now, we can describe K p : 

Proposition 5.4. Let (r, A) e A4. Let pq be the number of Xi 7^ 0, where A = 
(Ai, . . . , Xp'), andpi the number of distinct Xi ^ 0. We set A = (Ai, . . . , A po ) 6 W° . 
Let p £T f where f = rX* + D 2 (A). 

• If A = 0, then K p is the subgroup of K such that k.rX* — rX* for all 
k £ K p . 

• If A =/: 0, then K p is the direct product K± x K2, where: 

I he SO(2 Po ) \ 
I D 2 (k)k 1 = k 1 D 2 (A) j ' 

EK j k 2 .rX* p = rX p *| . 

Furthermore, K\ is isomorphic to the group K(m;po;pi) given by il(J\) . 

Proof. We keep the notations of this proposition, and we set A* = £>2(A) and 
X* = rX*. With Propositions 14.71 and the expression (TT5]) of the co-adjoint repre- 
sentation, it is easy to prove: 

(16) K p = {k € K : kA* = A*k and kX* = X*} . 

If A = 0, because of (16]), K p is the stability group in K of X* e V* - W. So the 
first part of Proposition 15.41 is proved. 

Let us show the second part. A ^ so we have 

HlJmi 



P Pl Jm P1 

where fi\, . . . , fi Pl are defined by (j4|), and m,j is the number of A^ = pj. We define 
m'j for j — 1, . . .pi by ((5]). 

Let k e K p . Because of (fT6|) . the matrices k and A* commute and we have: 

with ki € O(2p ) and k 2 € 0(p - 2p ) ; 

furthermore, by ^6|) . k 2 .X* = X*, and the matrices k\ and i?2(A*) commute. So 
fci is a diagonal block matrix, with blocks [fcij^ G 0(m,j) for i = l,...,pi. Each 





Id 





= 





k 2 



A* = 



' D 2 (A) 


' 









with D 2 (A) = 



ki 
k 2 
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block [fciL G 0{m,j) commutes with J mj . So on one hand we have det [ki]j = 1, 
det fci = 1, and on the other hand, [k\]^ G 0(rrij) corresponds to a unitary matrix 
([fa],-)- Now, we set for ki G A'i: 

= (^ roi) (Ni),---,4 m,,1 (N Pl )) • 

: A'i — ► K(m;po;pi) is a group isomorphism. □ 
As is an isomorphism which respects complexification, we have: 

Corollary 5.5. The isomorphism given during the previous proof respects com- 
plexification: 

i/j( po ' > {k 1 .(xi,yi,... ) x po) y Po )} = y 1 {k 1 ).ip( po) (x 1 ,y 1 ,...,x Po ,y Po ) . 

Quotient Group N — N / ker p. In this paragraph, we describe the quotient groups 
N/kerp and G/kerp, for some p G N. This will permit in the next paragraph 
to reduce the construction of the bounded spherical functions on N p to known 
questions on Euclidean and Heisenberg groups. For a representation p £ N, we will 
denote by: 

• ker p the kernel of p, 

• N — N/ ker p its quotient group and 77 its Lie algebra, 

• (%,/?) the induced representation on N, 

• n G N and Y G 77 the image of n <G N and Y E JV respectively by the 
canonical projections N — > N and Af — > 77. 

Now, with the help of the canonical basis, we choose the privileged representative 
p of T rX * + D 2 (A) as p = U rX ^D 2 (A) given by ©, with E 1 = RX 1 ffi . . .®RX 2po -i as 
maximal totally isotropic space for uJr) 2 (A),r- Because of Remark 14. 4| the quotient 
Lie algebra 77 has the natural basis: 



X 1 ,...,X 2po ,B = |A|- 1 C 2 (A) withA p ifr^O ; 

1 

here, we have denoted |A| = (£^ =1 A/) 2 = |-Da(A)| (for the Euclidean norm on 
Z). We compute that each Lie bracket of two vectors of this basis equals zero, 
except: 

[X 2 i_i,X 2 i] = rn- ' * = 1 :--->-Po ■ 

Let 77i be the Lie sub-algebra of 77, with basis Ai, . . . , X2 Po ,B, and let iVi be 
its corresponding connected simply connected nilpotent Lie group. We define the 
mapping * 2 : H Po — > ~N\ for h = (x 1 + iyi, . . . , x Po + iy Po ,t) £ BP by: 

9 2 (h) = exp { x 3 X ij-i + ViX2j) +iB 

Because of the values of the Lie brackets in N%, it is easy to see: 

Lemma 5.6. $2 is a group isomorphism between N± and HP . 

With our choice for representations and notations, we describe the induced rep- 
resentation and action: 



Proposition 5.7. With the notations set just above, 
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a) If A ^ 0, the two groups N and N% x N2 are isomorphic and the represen- 
tations ~p and ~pi ® are equivalent, where 

(1) p~Y * s a representation on N\ (whose expression may be computed); 

(2) N 2 and ~p 2 are described by: 

— either r = 0, then N 2 and ~p 2 are trivial; 

— or r ^ 0, then N 2 ~ RAT p , and p 2 ~ exp(xX p ) — » exp(ix). 

b) If A — 0, then N and ~p are the same as N 2 and ~p 2 above. 



Remark 5.8. Because of Remark \4-5[ the restriction of p 1 on the center expUB 
of M\ is given by: exp(a_B) exp(ia|A|). 

As K p is the i-T-stability group of p € N, it acts by automorphisms on N. Simple 
computations show that: 

Proposition 5.9. We keep the notations of Propositions \5.4\ and \5. 7[ 

a) If A = 0, K p acts trivially on N. 

b) //A^0, _ 

— K\ acts by automorphisms on N% and trivially on the center expRB 
ofNl, _ 

— K 2 acts trivially on N 2 . 

So the groups K p x N and {Ki x N\) x K 2 x N 2 are isomorphic. 

Recall Hheis — K (m; po; Pi) x M Po ; let us also define the group H = K\ x N\ 
and the map: 

j j Hh e i a — \ H 

Corollary [531 Proposition Ell and Lemma 15.61 imply: 

Proposition 5.10. "to is a group isomorphism between Hheis and H . 



Expression of (p v . Here we prove Theorem 15.31 Let p € N and v <E G p be fixed. 
We have z/ijv = c.p, f < c < 00, and we denote by v the induced representation on 
Kp x N. 

a) Case of the orbit 0(r, 0). We assumed that p — p r< Q. By Proposition l5.91 we have 
K p x N = K p x N. So V is the tensor product of an irreducible representation over 
N, which coincides with c.p (and so c = 1), with an irreducible representation over 
K p with a ifp-invariant vector, which is thus the trivial representation. We obtain 
that V coincides with (k, n) G K p x iVi-4 p(n). Now because of Proposition 15.71 b). 
v = v r ' is given by: 

(17) (k,exp(X + A)) ^— > e l<rX ? x> . 

So G p is the set of the classes of v r '° , where r ranges over M + , and we compute 
that the function (j> v for v = i/ r '° is given by ([H|: 

<f> v (n) = [ e l<rX ^ k - x> dk, n = exp(AT + A) £ N p . 

JkeK 
ir.O 



By ((3]), we have <j) v = 4> r '°, and Theorem 15.31 a) is proved. 
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b) Case of the orbit 0(r, A). We assume A ^ and p = p r ,A- For each bounded 
spherical function u> of the Gelfand pair (H Po , K(m;po;pi)), we define the repre- 
sentation of H such that : 

n^ = {Fo^- 1 , Fen^} and if = n w o y- 1 . 

By Proposition l5.9l b). v is the tensor product of three irreducible representations, of 
N 2 , H and K 2 , such that the vector space of K\ (respectively i^-uivariant vectors 
of the representations of H (respectively K 2 ) is a line. So the representation of K 2 
is trivial, and V induces a unitary irreducible representation V oi H x N 2 which 
coincides with c.p on N, and such that the vector space of K\ -invariant vectors is 
a line. 

We have v = 71 ® 72 where 

(a) 71 is an irreducible representation of H; the space of its i^i-invariant vectors 
is a line; it coincides with c.p over N\ ; 

(b) 72 is an irreducible representation over N 2 and coincides with c.p on N 2 . 

Because of irreducibility, (a) implies c = 1 and so (b) implies 72 ~ P2- Further- 
more, by Proposition 15.91 and Theorem 14.21 the irreducible representations on H 
such that the vector space of i^i -invariant vectors is a line, are all the representa- 
tions (Hu , II W ) , where w ranges over the set of bounded spherical functions of Hheis \ 
the ifi -invariant line of H" is Cfi^o^]/^ 1 . Thus, the representations 71 satisfying (a) 
are the representations such that 71 ~ IT" and ITj^ ~ Pi . Because of the expres- 
sions of 71 and II U on the center of iVi and H (see Remark l5.8l and equalities (TTTI) . 
(|12p) the case uj = us p is impossible if ~ p t . 

We have shown that V = 71 ® 72, where 72 ~ P2 is given in Proposition 15. 7\ and 71 
is among the representations equivalent to (H w ,n w ) with oj = uj\,i, I £ N Pl . 
Then i/ is among the representations equivalent to ("H" , i/ r ' A >') with = W|au> de- 
fined for n — exp(AT + A) £ N, and k — kik 2 £ K p where k\ £ Ki, and k 2 £ K 2 
by: 

(18) is r > A ' l {k,n) = e lr<x ?' X> Tl LU (k 1 ,q 1 (n)) , 
where q\ : N — > Ni is the canonical projection. 

We denote by G' p the set of classes of the representations i/ r - A ' 1 , I £ N Pl . A repre- 
sentation j/ r ' A '', has a unitary i^p-invariant vector u = fl u o ty^ . 
We still have to show that under formula (|T4)) . the function for v — v r - A ' 1 satisfies 
§ v = (j) r - A ' L . For n = exp(AT + A), it is given by: 

(19) (f> v {n) = [ e lr<X 'v k - X> uo^- 1 oq x {k.n)dk , 

Jk 

where uj — lo\a\,i- We compute 

u(I,n)u = e zr<x p' x> U u (I, qi(n))W o * , 
(v{I,n).u,u) n „ = e lr<x ' P ' X> (li^I^oq^n))^, Q")^ . 

As f2 w is the positive definite function associated to IT,, we have: 

(U^I^oq^n))^,^)^ = ^(I^^oq-^n)) 

= uj o ^>2 l <Ti (n) 
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We know the expression ui = uj\a\.i (Proposition ^. 8[) . and we compute those of * 2 1 
and <fi. Assuming (j4|), we obtain: 

9 r ' A ' i (exp(X + A)) = e ir<x 'v' x> Lu o ^ l o &(exp(X + A)) , 

where r > A >' is given by ([7]). Because of (fTOl) . ^ = <ff' A > 1 , and Theorem 15. 31 b) is 
consequently proved. 

The proof of Theorem 15.31 is now over. Theorem 13. II is thus proved. 

6. Representation over 7V p 

We give here the bounded spherical functions in terms of representations over 
N p [TJ Theorem G], which is an equivalent way for constructing them. 
We deduce then the eigenvalues of the sub-Laplacian for the bounded spherical 
functions and the expression of the radial Plancherel measure. 
Another expression of the bounded spherical functions <p v ' A ' 1 . Let us recall a few 
facts contained in pp. Let (N,K) be a Gelfand pair. Let (H,U) E N, and Kn its 
instability group. There exists a projective representation Wn of Kjj on 'H, and 
an orthogonal decomposition of % — Vl m to irreducible subspaces Wn -invariant. 
For C £ T-L, let us define the function (j)n£ by: 

<fa,c(n) = / < n(fe.n)C, C > dk , n€N . 

The spherical functions are the </>n,c f° r C S Vi, |£| = 1 and II € N; the spherical 
function <fin^ is independent of ( € Vi, |C| = 1 and of the choice of the representative 
ofn. 

Remark 6.1. Furthermore, for a K -invariant function f on N and a K left- 
invariant differential operator D on N, Tl(f) and dH(D) are Wn-invariant; they 
equal the identity on each Vi up to a constant; and for Vi , and the corresponding 
spherical function <f>, the constants are respectively </,</>> and the eigenvalue of 
D for (f>. 

Now, we give the link between the given spherical functions and the above de- 
scription for each class of N p , which corresponds to a class 0(X* , A*). 
Before this, we recall definitions and properties of Hermite functions. We denote 
by H k the Hermite polynomial: 

H k (s) = (-l) k e s2 (d/ds) k e- s2 , 
by hk, k £ N the Hermite functions on R 12 s §5.5]: 

h k (x) = {2 k k\^y~ 2 e-^H k {x) , 

and by h a ,a £ N™ the Hermite functions on R": h a = H™ =1 h ai . We recall that 
the Hermite functions h k , k € N on R form an orthonormal basis L 2 (R). 
Case A* = 0. p = p r ,0 is the one-dimensional representation given by 



exp(A + A) H> cxp(z < rX*,X >) 
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Case A* ^ 0. Let (r,A) £ .M with A ^ ct 1 e N Pl . Let £ ; be the set of 
a = (a 1 , . . . , a Pl ) where a j = {a\ ) , <i< , £ W' l] such that 

3 — 1 — J 

= "i=^ for j = . 

m' . <z<m' 
3 — 1 j 

Let us define the representation (H, II) = (L 2 (R Po ), IL^a) € N p for / € (y 1: . . . , y po ) £ 
RP° , n = exp(X + A) G 7V p by: 

II(n)./(y) = e l< - D 2(A)^>+i<rX*,X>+iX;f =1 ^-x 2] x 2j - 1 +^ r >T J x 2j y ] 
fiVl + VMxi, . ■ . ,y Po + y/Xp^X 2po -l) , 

where X — Y%=i x jXj- 

It is easy to see that the representation II is equivalent to p r ,A- So II g T rX *+D 2 (A)> 
II is irreducible, and its if-stability group denoted K n equals K p . 
Let Cq € H, a £ E[ be given by: 

( W> — > K 

(a ■ I Pl 

\ 3 = 1 3-1 J 

The vectors ( a ,a £ Ei,l £ N Pl form an orthonormal basis of H . 

It can be proved that each vector space Vi generated hy a £ Ei is ifn-invariant. 

Using relation between Laguerre and Hcrmite functions, we obtain: 



Lemma 6.2. The spherical function associated to II r ,A and V\ is <jf' < ' . 
Consequences for Sub-Laplacian. The (Kohn) sub-Laplacian is 

p 

L:=-^Xf . 
i=i 

It is a sub-elliptic if-invariant operator (with analytic coefficients). Consequently 
the spherical functions are eigenfunctions for this operator (see Remark 14. ip . 
Each representation II = II r ,A induces the representation dM on the algebra of 
differential left invariant operators on N p , over the space of Schwartz functions 



j = l,...,p 


dU(X 2j _ 1 ) 


- v^jdyj 


j = l,...,p 


dIL(X 2j ) 


= i \fhVi 


2po < j <P 


dn{x 3 ) 


= 


j =P 


dn{x p ) 


= irld 


Vi < j 


<m(x itj ) 


f iXjdd 



if (i,j) = (2f-l,2f) 
otherwise 



For L, we get: 

Po 

dU(L) = r 2 Id-J2^K-y!) 

i=l 



17 



We deduce (see Remark ETT 



We recall the Hermite function y = hk satisfies the differential equation y" + (2k - 
1 - x 2 )y = [T2J formula (5.5.2)]. So, we get: 

pi 

dn(L).( a = \ J2 X j( 2l j + m o) + r2 | Ca , a e E u I € N Pl 



L#' A '< - f£A i (2l i +m i ) + r a » ^■ A ' 1 

this equality may also be computed directly using properties of the Laguerre func- 
tions. 

Radial Plancherel measure. Here, we give the radial Plancherel measure. 

Let C be the set of A = (Ai, . . . , A p <) € W' such that Ai > . . . > \ p > > 0. We 
define the following measure on C: 

• dA = dXi . . . dX p i is the restricted Lebesgue measure on C, 

• 1] is the measure on C such that: 



dr)(A) 



cn j<fc (A2-A2) 2 rfA ifp = 2j/ 

clhXllL^kiX] -X 2 k fdA ifp = 2p' + i 



where the constant c is chosen in order to yield the polar change of variables 
over the space of antisymmetric matrices A p : 

g(A)dA = f [ g(k.D 2 {A))dr)(A)dk . 
Jo p J c 

• rj is the measure on C given by: drj'(A) = H^ =1 Xidrj(A) , 

Over M + , we define the measure r given as the Lebesgue measure if p = 2p' + 1, 
and the Dirac measure in if p — 2p' . 

The (non radial) Plancherel measure is already known [TTJ Section 6] . With our 
notations, it is the measure m given as the tensor product of the Haar probability 
measure dk on K = 0(p), the measure rf on C, and the measure r on K + , up to 
the constant c(p) given by: 



c(p) = 




ifp = 2p>, 
Hp = 2p' + l 



Theorem 6.3. m is the (non radial) Plancherel measure; i.e. for ip € L 2 (N), we 
have: 

IHIi3(;v) = J \\k.U rtA (iP)\\ 2 HS dm(r,k 7 A) . 
where \\-\\ H g denotes the Hilbert- Schmidt norm. 

If we compute the Hilbert-Schmidt square-norm of kM r ^(il)) with the orthonor- 
mal basis {(, a ,ct £ Ei,l £ N Pl }, we deduce the radial Plancherel measure (see 
Lemma 16. 2|) . This is the measure, which we denote by m\ given as the tensor 
product of 77' on £, and the counting measure on N p , and the measure r on R+, 
up to the normalizing constant c(p): 
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Theorem 6.4. rrfi is the radial Plancherel measure for (N P7 O p ), i.e. for a K- 
invariant function tp G L 2 {N), we have: 

U\\l HN ) = / I < V>> 4> r ' A ' 1 > \ 2 dm\r,A,l) . 

We can also compute directly the radial Plancherel measure m) > , using the prop- 
erties of Laguerre functions and Euclidean Fourier transform (see [5]). 
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